Subject review judgements and reports

Judgements on academic standards

The judgement on standards is a single, threshold judgement, of confidence or otherwise in standards, made after considering each of the following components:

• appropriateness of the intended learning outcomes
• effectiveness of curriculum content and design, including effectiveness in securing academic and intellectual progression
• appropriateness of assessment in measuring the intended outcomes; security and integrity of the assessment process
• actual student achievement in relation to the intended outcomes and the level of the award

Other points about the standards judgement:

• judgements are not graded – according to the QAA either intended learning outcomes are achieved or they are not
• if standards are being achieved but reviewers are concerned that they may not be maintained, a judgement of ‘limited confidence’ may be made
• if failure/success in achieving standards has occurred in programmes at one level only, the failing/succeeding level will be identified separately
• where a judgement of ‘limited confidence’ in academic standards is made, subject reviewers must identify areas where improvement is needed
• the department/school may then be asked to prepare an improvement strategy
• if a judgement is made that standards are not being achieved there will be a further formal review within one calendar year

It is not anticipated that many judgements of ‘limited confidence’ in academic standards will be reported.

Judgements on quality of learning opportunities

Judgements are made about the extent to which the following three aspects of provision contribute to the achievement of intended learning outcomes.

Teaching and learning:

Effectiveness of delivery, in relation to curriculum content and programme aims, through:

• large and small group teaching
• practical sessions
• directed individual learning
• integration of skills within curricula
• distance learning

Student progression:

• recruitment, including matching intake to programme requirements
• academic support, including tutorial arrangements and feedback to students
• progression within the programme and wastage

Learning resources:

Effective utilisation of:

• equipment, including IT
• accommodation
• library, including electronic resources
• staff, including academic, administrative and technical

Reporting on the quality of learning opportunities will place each of the three aspects of provision into one of three categories, ‘failing’, ‘approved’ or ‘commendable’, and will be made on the following basis:

• provision makes a less than adequate contribution to the achievement of the intended outcomes. Significant improvement is required urgently if the provision is to become at least adequate. In the summary report, this judgement will be referred to as ‘failing’.
• provision enables the intended outcomes to be achieved, but improvement is needed to overcome weaknesses. In the summary report, this judgement will be referred to as ‘approved’. The summary will normally include a statement containing the phrase ‘approved, but…’, which will set out the areas where improvement is needed.
• provision contributes substantially to the achievement of the intended outcomes, with most elements demonstrating good practice. In the summary report, this judgement will be referred to as ‘commendable’.

Within the ‘commendable’ category, reviewers will identify any specific features of the aspect of provision that are exemplary. To be deemed ‘exemplary’, a feature must:

• represent sector-leading best practice
• be worthy of dissemination to, and emulation by, other providers of comparable programmes
• make a significant contribution to the success of the provision being assessed

Incidental or marginal features do not qualify for designation.

At the end of each subject review a 4,000 word report is published comprising a brief description of the review method, the overall aims of the subject provider, an evaluation of the quality of learning opportunities and the academic standards achieved, the conclusions reached and judgements made and a one-page summary of conclusions.

Subject reviewers will gather evidence through discussions with staff and students and scrutiny of examiners’ reports on how institutional systems operate in each subject area.

The final section of the subject report will express their level of confidence in the institution’s ability to maintain quality and standards and these views will inform the institutional review.

The ‘exemplary’ rating

QAA has recently issued the following additional information on the ‘exemplary’ descriptor (more information is provided in the appendices).

“Exemplary features will not be common. They relate to a specific feature of an aspect of provision. “Exemplary” is not a fourth category of judgement, above the “commendable” category. To be deemed “exemplary” a feature must satisfy all three of the criteria listed below.
The feature must:

• represent sector-leading best practice; and
• be worthy of dissemination to, and emulation by, other providers of comparable programmes; and
• make a significant contribution to the provision being assessed.
  “Exemplary” signifies not just excellence, but excellence which is generalisable and transferable.”

Additional details

Subsequent to the publication of the its handbook for academic review the QAA has (November 2000) issued further advice on judgements which is reproduced here.

“With respect, first of all, to academic standards, it is clear that the normal expectation will be to make a judgement on the overall standards prevailing for the subject under scrutiny. There may, however, be situations where the standards achieved by, for example, Honours degree students are high but for sub-degree work or postgraduate work they are not. Here it may be necessary to differentiate between programmes at different levels in the narrative section of the subject review report on standards, and possibly in the judgements.

In view of the fact that the overall judgements are being made at the level of the subject (and not the programme), it is at this point that the reviewers must decide whether the extent of the provision in which they do not have confidence is sufficient to warrant an overall ‘no confidence’ judgement. On the other hand, it may be appropriate to express confidence in the overall standards but to draw attention in the narrative of the report to a minority of programmes in which the academic standards achieved are lower than they should be.

In some cases, it may thus be necessary to make judgements of confidence in the standards for some programmes but not for others. In reality, such cases are likely to be rare, but reviewers must be prepared to differentiate where they believe that an overall judgement would obscure important issues.

Secondly, as far as aspects of quality are concerned, the situation is potentially even more complicated because of the possibility of aggregation of subjects. Again, judgements will normally cover all provision within the scope of the review, but if performance is significantly different in a subject area, or at a particular level within a subject, separate judgements will have to be made.

Reviewers will have to decide whether the extent of the weaknesses observed is sufficient to lower the overall judgement of the aspect of provision to ‘approved but’ or ‘failing’. It is highly unlikely that an overall judgement of ‘commendable’ could be made in circumstances where one subject of those aggregated shows weaknesses, but it is just conceivable that otherwise commendable provision could show small weaknesses at one level within a subject.

The final grading for each aspect thus remains very much a matter for reviewer judgement, taking the full extent of the provision and the weaknesses identified into consideration. Even if the overall final judgement is ‘commendable’, the QAA would expect all weaknesses to be clearly identified within the report narrative on the aspect of provision in question.”

— (QAA letter to institutions from Peter Milton, November 2000)

Further guidance on making judgements is provided to reviewers as part of their training by the QAA.

Reports

The review coordinator produces the first draft of the report immediately after completion of the review, drawing on the self-evaluation and on the summaries prepared by subject reviewers. This draft is then checked by reviewers for factual accuracy and affords an opportunity for further comment before the report is despatched to the institution.

The published reports are the main documented outcomes of the subject review process. Publication should take place within 20 weeks from the end of a review. The QAA expects that reports will be characterised by succinct, accurate writing and a clear, consistent style. The evidence base must be sound, and must be recorded accurately by reviewers.

About six to eight weeks after the visit you should receive a draft report. You are given an opportunity to comment on the factual accuracy of the report and on any misconceptions or misrepresentations it contains.

The following points should be borne in mind:

• the points identified by reviewers in the report will need to be addressed via annual monitoring/review and periodic re-validation/programme review
• it would be a great waste of effort and time if the procedures, knowledge, practices and indeed the enthusiasm for teaching and learning issues generated by a subject review were to be lost in a few months. Maintaining the momentum will ensure that the department/school will not only retain the capacity to respond to external scrutiny but it will also ensure that staff and students will derive considerable educational benefit.
• you should ensure continuing review of the subject/departmental aims, objectives, curriculum etc (all the things covered by the review methodology)
• the currency of the documentary evidence gathered for subject review should be maintained and someone should be given the responsibility to ensure this happens
• staff and students should be kept aware of their roles in quality procedures
• any new or improved links established with central services should be sustained

Subject review will continue in one form or another. Maintaining the impetus developed before and during the visit will not only make future reviews considerably easier to prepare for it will positively contribute to the maintenance and enhancement of the quality of provision in your department/school.

And finally…

Subject review matters. It offers a genuine opportunity for departments/schools to achieve deserved recognition for the quality of their activities and the chance to reflect on and enhance the student learning experience. You will naturally wish to achieve the high grades that reflect the high quality of teaching and learning you offer. Good luck!

Last Modified: 30 June 2010